Setup command line client

The following is code to get the client and set it up.



In [3]:

    
%env CLIENT bitrepository-client-1.9-RC1

!wget -Nq "https://sbforge.org/download/attachments/25395346/${CLIENT}.zip"
!unzip -quo ${CLIENT}.zip

%alias bitmag ${CLIENT}/bin/bitmag.sh %l









    



env: CLIENT=bitrepository-client-1.9-RC1



In [4]:

    
#Some imports we will need later
import random
import string

Describes the tests needed to validate the PutFile functionality.

The commands presented are just examples of what the actual command to test with could look like.

Basic put

Put a file to all pillars

Run put command on a file which is not present in the collection.



In [5]:

    
TESTFILE1='README.md'

Check this by running



In [6]:

    
%bitmag get-file-ids -c integrationtest2 -i {TESTFILE1}









    



Count: 	Size: 	FileID:
5 	4141 	README.md

The response should be

Count:  Size:   FileID:

Indicating that no pillar had the file. If the file is found, delete it with the following command

See https://www.safaribooksonline.com/blog/2014/02/12/using-shell-commands-effectively-ipython/ for how this works



In [7]:

    
hash=!cat {TESTFILE1} | md5sum - | cut -d' ' -f1
print("md5: {}".format(hash))

%bitmag delete -c integrationtest2 -i {TESTFILE1} -C {hash.s} -p "sbtape2"
%bitmag delete -c integrationtest2 -i {TESTFILE1} -C {hash.s}  -p "reference2"
%bitmag delete -c integrationtest2 -i {TESTFILE1} -C {hash.s}  -p "checksum2"
%bitmag delete -c integrationtest2 -i {TESTFILE1} -C {hash.s}  -p "sbdisk1"
%bitmag delete -c integrationtest2 -i {TESTFILE1} -C {hash.s}  -p "kbpillar2"









    



md5: ['e3815360b4db52d61191d985b79a208c']



In [8]:

    
%bitmag put-file -c integrationtest2 -f {TESTFILE1}

Verify that it is now present at all pillars, and that it has the same checksum at all pillars



In [9]:

    
%bitmag get-file-ids -c integrationtest2 -i {TESTFILE1}









    



Count: 	Size: 	FileID:
5 	4141 	README.md



In [10]:

    
%bitmag get-checksums -c integrationtest2 -i {TESTFILE1}









    



Count: 	Checksum: 	Pillars: 	FileID: 
5 	e3815360b4db52d61191d985b79a208c 	All 	README.md

Put a file with a different file-id to all pillars

Run put command on a file and use the file-id parameter (-i)



In [11]:

    
TESTFILE2=''.join(random.choice(string.ascii_uppercase + string.digits) for _ in range(10))



In [12]:

    
%bitmag put-file -c integrationtest2 -f {TESTFILE1} -i {TESTFILE2}

Verify that it is now present at all pillars, and that it has the same checksum at all pillars



In [13]:

    
%bitmag get-file-ids -c integrationtest2 -i {TESTFILE2}









    



Count: 	Size: 	FileID:
5 	4141 	VI3MM22C2P



In [14]:

    
%bitmag get-checksums -c integrationtest2 -i {TESTFILE2}









    



Count: 	Checksum: 	Pillars: 	FileID: 
5 	e3815360b4db52d61191d985b79a208c 	All 	VI3MM22C2P

Put a file by using an URL

Use the URL and Checksum to put a file, already on the webdav server.



In [15]:

    
TESTFILE3=''.join(random.choice(string.ascii_uppercase + string.digits) for _ in range(10))
URL='http://sandkasse-01.kb.dk/dav/test.txt'



In [16]:

    
hash=!curl -s {URL} | md5sum - | cut -d' ' -f1
print("md5: {}".format(hash))
%bitmag put-file -c integrationtest2 -u {URL} -i {TESTFILE3} -C {hash.s}









    



md5: ['940a51b250e7aa82d8e8ea31217ff267']

Verify that it is now present at al pillars, with the requested checksum



In [17]:

    
%bitmag get-file-ids -c integrationtest2 -i {TESTFILE3}









    



Count: 	Size: 	FileID:
5 	27 	18A58S6PNM



In [18]:

    
%bitmag get-checksums -c integrationtest2 -i {TESTFILE3}









    



Count: 	Checksum: 	Pillars: 	FileID: 
5 	940a51b250e7aa82d8e8ea31217ff267 	All 	18A58S6PNM

Idempotent test

Attempt to put a file with an fileID that is already present in the collection suppling the same checksum

The putFile request should succeed.



In [19]:

    
%bitmag put-file -c integrationtest2 -f {TESTFILE1}

Put with returned checksums

Put a file including a request of a salted checksum calculated on the pillar, which should be returned.



In [20]:

    
import hmac
import hashlib
import urllib.request

def getSaltedChecksum(url,salt,algorithm):
    saltBytes = bytes.fromhex(salt)
    digester = hmac.new(saltBytes,None,algorithm)
    with urllib.request.urlopen(url) as from_fh:
        while True:
            chunk = from_fh.read()
            if not chunk:
                break
            digester.update(chunk)
    return digester.hexdigest().lower()

saltedChecksum=getSaltedChecksum(url=URL,salt='abcd',algorithm=hashlib.sha1)
print(saltedChecksum)









    



5bb8cf7554e747f7ab8bd09a910b680bf889d959

Use the request-checksum-algorithm and request-checksum-salt arguments (-R and -S):



In [21]:

    
TESTFILE4=''.join(random.choice(string.ascii_uppercase + string.digits) for _ in range(10))

Note, this will return the stored MD5 hashes, if the file is already put'et



In [22]:

    
hash=!curl -s {URL} | md5sum - | cut -d' ' -f1
print("md5: {}".format(hash))
%bitmag put-file -c integrationtest2 -u {URL} -i {TESTFILE4} -C {hash.s} -S 'abcd' -R HMAC_SHA1









    



md5: ['940a51b250e7aa82d8e8ea31217ff267']
PillarId 	 Checksum
sbtape2 	 5bb8cf7554e747f7ab8bd09a910b680bf889d959
reference2 	 5bb8cf7554e747f7ab8bd09a910b680bf889d959
sbdisk1 	 5bb8cf7554e747f7ab8bd09a910b680bf889d959
Failure: 94a781d1: PUT_FILE for file IZOX164686: FAILED: , Failed operation. Cause(s):
[94a781d1: PUT_FILE for file IZOX164686: COMPONENT_FAILED: ContributorID kbpillar2 ResponseCode: FILE_TRANSFER_FAILURE, File transfer for IZOX164686 to http://sandkasse-01.kb.dk/dav/test.txt]

Verify that the checksumpillar does not reply with a checksum.



In [23]:

    
%bitmag get-checksums -c integrationtest2 -i {TESTFILE4} -R HMAC_SHA1 -S 'abcd'









    



Count: 	Checksum: 	Pillars: 	FileID: 
3 	5bb8cf7554e747f7ab8bd09a910b680bf889d959 	All 	IZOX164686

Verify that the checksumpillar and all the other pillars have the file



In [24]:

    
%bitmag get-file-ids -c integrationtest2 -i {TESTFILE4}









    



Count: 	Size: 	FileID:
4 	27 	IZOX164686

Put a file including a request for checksum, but using both a salt and a non-salt algorithm.

Use the request-checksum-algorithm and request-checksum-salt arguments (-R and -S):



In [25]:

    
TESTFILE5=''.join(random.choice(string.ascii_uppercase + string.digits) for _ in range(10))



In [26]:

    
hash=!curl -s {URL} | md5sum - | cut -d' ' -f1
print("md5: {}".format(hash))

%bitmag put-file -c integrationtest2 -u {URL} -i {TESTFILE5} -C {hash.s} -R SHA1 -S 'abcd'









    



md5: ['940a51b250e7aa82d8e8ea31217ff267']
PillarId 	 Checksum
kbpillar2 	 5bb8cf7554e747f7ab8bd09a910b680bf889d959
sbtape2 	 5bb8cf7554e747f7ab8bd09a910b680bf889d959
reference2 	 5bb8cf7554e747f7ab8bd09a910b680bf889d959
sbdisk1 	 5bb8cf7554e747f7ab8bd09a910b680bf889d959

Verify that all the data pillars return the same checksum, and that it is identical to the previous test (where '-R HMAC_SHA1')

Put a file including a request for checksum, but using a salt algorithm but not giving a salt.

Use the request-checksum-algorithm and request-checksum-salt arguments (-R and no -S):



In [27]:

    
hash=!curl -s {URL} | md5sum - | cut -d' ' -f1
print("md5: {}".format(hash))

hash=!curl -s {URL} | sha1sum - | cut -d' ' -f1
print("sha1: {}".format(hash))

%bitmag get-checksums -c integrationtest2 -i {TESTFILE5}









    



md5: ['940a51b250e7aa82d8e8ea31217ff267']
sha1: ['82b4b156a5db351e278e3a8e5d3e762deca45fdb']
Count: 	Checksum: 	Pillars: 	FileID: 
5 	940a51b250e7aa82d8e8ea31217ff267 	All 	TRK40HUT3Q

Verify that the data pillars all deliver the same checksum, which must be different from the checksum in the previous two tests.

Error scenarios

Attempt to put a file with an fileID that is already present in the collection (ensure that the file has a different checksum from the already archived)

The putFile request should fail informing the user that a file with the given file ID already exists in the collection.



In [28]:

    
%bitmag put-file -c integrationtest2 -f .gitignore -i {TESTFILE1}









    



Failure: ecf3f31d: PUT_FILE for file README.md: FAILED: , Can not put file README.md, as an different file already exists on pillar kbpillar2

Attempt to put a file which does not exist

Client should fail immediately



In [29]:

    
%bitmag put-file -c integrationtest2 -f ThisFileDoesNotExist









    



java.lang.IllegalArgumentException: The file 'ThisFileDoesNotExist' is invalid. It does not exists or it is a directory.
	at org.bitrepository.commandline.CommandLineClient.findTheFile(CommandLineClient.java:283)
	at org.bitrepository.commandline.CommandLineClient.getURLOrUploadFile(CommandLineClient.java:329)
	at org.bitrepository.commandline.PutFileCmd.putTheFile(PutFileCmd.java:159)
	at org.bitrepository.commandline.PutFileCmd.performOperation(PutFileCmd.java:86)
	at org.bitrepository.commandline.CommandLineClient.runCommand(CommandLineClient.java:68)
	at org.bitrepository.commandline.PutFileCmd.main(PutFileCmd.java:50)

Attempt to put a file to a non-existing collection

Client should fail immediately



In [30]:

    
%bitmag put-file -c integrationtest3 -i {TESTFILE1}









    



Invalid argument: integrationtest3 is not a valid collection.
The following collections are defined: [integrationtest1, integrationtest2, reklamefilm, dvd, avis, netarkiv]

Attempt to put a file to a non-existing pillar

Client should fail immediately



In [31]:

    
%bitmag put-file -c integrationtest1 -i {TESTFILE1} -p non-existing-pillar









    



Invalid argument: non-existing-pillar is not a valid pillar for collection integrationtest1
The collection contains the following pillars: [sbtape2, reference2, checksum2, sbdisk1, kbpillar2]